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(Furco & Moely, 2012, pg. 129). This paper reports the results of an exploration of factors that influence 
faculty to consider or reject using analysis of student data to improve instruction based on social cognitive 
theory. Self-efficacy, value of the outcome, and feasibility of using a student data-based reflection process were 
found to be related to the actual use of components of the reflection process by faculty. 
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Much is being said in education about the value of adopting data-based or analytics approaches to instructional improve¬ 
ment. One important group of stakeholders in this effort is the faculty.“ln many cases, the key constituency group is facul¬ 
ty, whose powerful voice and genuine participation often determine the success or failure of educational innovations, es¬ 
pecially those that involve pedagogical and academic change” (Furco & Moely, 2012, pg. 129). This paper reports the 
results of an exploration of factors that influence faculty to consider or reject using analysis of student data to improve 
instruction based on social cognitive theory. Self-efficacy, value of the outcome, and feasibility of using a student da¬ 
ta-based reflection process were found to be related to the actual use of components of the reflection process by faculty. 


INTRODUCTION 

Much is being said in education about the value of adopting data- 
based or analytics approaches to instructional improvement. 
Writing about the rise of analytics as the vanguard of this approach, 
Campbell, DuBois and Oblinger (2007) said,“Whether the catalyst 
for adoption is a call for accountability from outside of higher 
education or the need for scorecards or decision-making models 
from within, analytics is in higher education’s future” (pg. 41). 

One important group of stakeholders in this effort is the 
faculty.“ln many cases, the key constituency group is faculty, whose 
powerful voice and genuine participation often determine the 
success or failure of educational innovations, especially those that 
involve pedagogical and academic change” (Furco & Moely, 2012, 
pg. 129).This paper reports the results of an exploration of factors 
that influence faculty to consider or reject analysis of student data 
to improve instruction. 

To what degree are faculty willing to base the success or 
failure of their teaching on student data? In a survey of faculty 
trust in the accuracy of learning analytics (Drachsler & Greller, 
2012), responses fell halfway between no confidence and total 
confidence. The authors attributed their findings to faculty having 
“a slight skepticism toward ‘calculating’ education and learning.” (pg. 
7) In this paper, we discuss how interest in student data-centered 
models for instructional improvement has surfaced under different 
names and different theories of instructional improvement and the 
role of faculty in its progress. 

Early Efforts to Adopt a Student Data-based Model 
for Instructional Improvement 

In the early ‘90’s the idea that instructional improvement should 
be based on verifiable data was adopted by leaders in the faculty 
development. Individuals like K.Patricia Cross, Thomas Angelo, 
Wilbert McKeachie, Art Chickering, Zelda Gamson, and many 
others looked for ways of encouraging faculty to be more systematic 
in their teaching. The Classroom Assessment Techniques and 
Classroom Research movement Cross and Angelo championed 
was a turning point in this direction at the university level. 

Classroom Assessment Techniques. Attempts to adopt 
instructional improvement based on student data were encouraged 
by the work of Angelo and Cross (1993). These authors inspired 
faculty to gather data about learning by offering classroom 
assessment techniques (CATs) that could be used easily in classes. 
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The techniques included activities such as the Minute Paper, the 
Muddiest Point in the day’s class, and concept mapping to determine 
how well students understood class that day. The CATs were very 
popular with faculty and still are widely used to monitor student 
learning. 

Classroom Research/Scholarship of Teaching and 
Learning. Cross subsequently introduced the idea of engaging 
in Classroom Research, a more teacher driven version of action 
research that was common in education (Cross and Steadman, 
1996; Angelo, 1998). Classroom Research was an early version of 
the Scholarship ofTeaching and Learning (SOTL) movement (Huber 
& Hutchings, 2005; Kreber, 2007). The biggest difference between 
the two strategies was that Classroom Research was focused more 
on understanding a particular class situation and not on creating 
a literature base for teaching and learning in higher education. 
SOTL and various instantiations were focused on applying practical 
research strategies to find more effective learning. SOTL aimed 
also to create a field of research and a body of literature to support 
instructional improvement. 

Classroom Research and SOTL both inspired faculty by these 
activities. While Classroom Research has continued to be done 
by individual faculty in their classes, SOTL has founded scholarly 
journals, and inspired communities of inquiry as faculty find others 
with similar questions about teaching. The Carnegie Foundation 
for the Advancement ofTeaching has been especially instrumental 
in nurturing this format of communities across disciplines for 
investigating student learning in real classrooms. 

Learning Analytics. The enthusiasm faculty exhibited for 
CATs and SOTL has not yet generalized to using the kind of “big” 
data that many administrators and accreditors prefer (Andrade, 
2011; Siemens & Long, 201 I ).These data, called “academic analytics” 
(Campbell, DeBlois & Oblinger, 2007) and done on databases of 
information available through technology, are viewed with some 
skepticism by faculty (Parry, 2012). This technology-based data 
usage has made more inroads with faculty when the focus is on 
“learning analytics”, directed more at student learning in a context 
(Siemens & Long, 201 I). These analyses are more systematic than 
Classroom Research studies, but not based on large numbers of 
students like the “academic analytics.” They are closer to action 
research, although their questions differ. According to Dyckhoff, 
Lukarov, Muslim, Chatti, and Schroeder (2013), action research 
derives from teacher questions, whereas learning analytics come 
more from close analysis of data already collected. Dyckhoff, et al. 



(201 3) reported that the types of data typically gathered by learning 
analytics were not able yet to answer most questions that teachers 
had. They argued for teachers to work with analysts to shape 
indicators and collection methods tied to teachers’ questions. 

Faculty cooperation in gathering and interpreting information 
about learning is key to the success of all such efforts. Therefore 
we wonder why some approaches to data use like CATs and SOTL 
spark interest, while others like Academic Analytics, are met with 
suspicion or skepticism. Macfadyen and Dawson (2012) conducted 
a study of the use of the learning management system at one 
university and concluded that technical rather than teaching and 
learning issues were often the focus of administrative decisions 
about data. The authors concluded that “to have meaningful 
impact, learning analytics proponents must also delve into the 
socio-technical sphere to ensure that learning analytics data are 
presented to those involved in strategic institutional planning in 
ways that have the power to motivate organizational adoption 
and cultural change" (pg. 149). A first step to increasing faculty 
enthusiasm for student data for improvement would be to apply 
theories of behavior change from other fields, specifically social 
cognitive theories, to understand faculty beliefs about when, how 
and why they already gather and use student data and how it could 
be more useful.This was the focus of the current study. 

A Model Emphasizing Factors Affecting Faculty 
Use of Student Data 

In Figure I we provide a model of what factors we chose to 
investigate in this study. We will refer to this as the Factors model 
when discussing it in the text.The factors have been drawn from 
the literature on motivation for change in many contexts and from 
literature on how instructors come to try innovations . Much of 
the literature on these topics has been generated in K-12 education, 
in technology-based education, and especially in health behavior 
studies. Despite this variety of contexts, we believe that the same 
forces operate in higher education settings. 

First we show a composite model (the Factors model in 
Figure I) that illustrates some of the factors that the literature leads 
us to believe will affect the acceptance of innovations in student 
data collection and use. We highlight theories on individual choice 
and provide brief overviews on each theory and its relevance to 
faculty decisions to innovate. We then summarize and relate our 
findings to research on faculty use of student data. 

Theoretical Perspectives on Factors Influencing 
Faculty Use of Student Data 

Since faculty are the ones closest to attempts to change instruction, 
understanding their position is a critical factor in expanding change 
based on the use of student data. Researchers had to understand 
faculty current beliefs about student data and learning analytics, 
in order to convince faculty that studying such data would be 
worthwhile (Dyckhoff, 201 I). 

We hypothesized that major factors influencing faculty to use 
data were their attitudes and beliefs (see Figure l).We drew on 
current theories of motivation for behavior change in education 
innovation and health promotion grounded in the educational and 
social psychology literature. We selected the following factors as 
possible keys to adoption of student data use: 

Teacher self-efficacy for student data gathering and use, 
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Teacher beliefs about the value of student data, 

Teacher beliefs about the feasibility of making changes in 
their personal and institutional context, 

Teacher beliefs about the effort required to use data for 
change. 

Favorable values for all these beliefs could lead to positive 
attitudes about using data for instructional improvement. These 
factors were drawn from the following social cognitive theories 
about motivation in general and for innovation and behavior change: 
Self-efficacy component of Social Cognitive Theory 
(Bandura, 1986) 

Expectancy Value theory of motivation (Wigfield and 
Eccles, 2000) 

Self-determination theory of motivation (Deci and Ryan, 

2000 ) 

Theory of Planned Behavior (Madden, Ellen & Ajzen, 1992) 
Adoption or Diffusion of Innovations Theories (Rogers, 
2003) 

These theories came from different domains and addressed 
special concerns within those domains, and yet have much in 
common, as asserted by Conner and Norman (2005a) about 
health behaviors. We reviewed Connor and Norman’s attempt to 
synthesize a more integrated theory and its potential applications 
to faculty decisions to innovate in the discussion section of this 
paper. Here we highlight some of the key common factors. 

Factor I: Faculty Self-efficacy for Collecting and 
Using Student Data. 

Self-efficacy. The first factor included in the Factors model was 
a teacher’s self-efficacy for the collection and use of student data. 
Self-efficacy in this context is defined as instructors’ belief in their 
own ability to successfully gather and interpret student data for 
improving instruction. Variations of this belief in one’s capability 
to be successful at a specific behavior are found in almost every 
theory of innovation adoption. Bandura (1986) identified self- 
efficacy as a key component of social cognitive theory. Self-efficacy 
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has been shown to be important in motivation and performance in 
a variety of contexts (Klassen,Tze, Betts & Gordon, 2011; Pajares, 
1996; Schunk & Pajares, 2009; Van Acker, van Buuren, Kreijns, & 
Vermeulen, 201 3; Wigfield,Tonks, & Klauda, 2009). 

In a chapter about the expansion and acceptance of self- 
efficacy in Social Cognitive Theory, Luszczynska and Schwarzer 
(2005) offered the following observation on its importance. 
...self-efficacy models are no longer really distinct from 
other approaches because the key construct that was 
originally development within Bandura’s social cognitive 
theory has subsequently proved to be an essential 
component of all major models, (pg. 144) 

The role of self-efficacy in teaching has been explored most 
widely in the K-12 system using the Teachers’ Sense of Efficacy 
scale developed by Tschannen-Moran, Hoy, and Woolfolk-Hoy 
(2001). In research on the scale’s model, Tschannen-Moran, 
Woolfolk-Hoy, and Hoy (1998) found efficacy beliefs predicted 
teachers’ goal selection, effort expended, and persistence. In 
another study of the role of self-efficacy in teacher behavior at the 
K-12 level, Van Acker, van Buuren, Kreijns, and Vermeulen (2013) 
found that teacher attitudes toward technology and self-efficacy 
for technology use were the top influences on their use of digital 
learning materials in teaching. The spread of such studies increased 
with the growing acceptance of technology for teaching (Holden & 
Rada, 201 I). Reviews of self-efficacy research in K-12 teachers have 
been increasingly instrumental in encouraging teacher education 
programs to be mindful about how self-efficacy affects a teacher’s 
development (Woolfolk-Hoy, Davis & Pape, 2006). 

There is not yet a similar extensive analysis of self-efficacy in 
postsecondary faculty, except in the area of technology use. More 
work has been done internationally than in the United States. 
Examples of research involving postsecondary teachers include a 
study in Taiwan by Chang, Lin, and Song (201 I), research by Norton, 
Richardson, Hartley, Newstead and Mayes in the UK (2005), by 
Prieto Navarro (2005), and Vera, Salanova and Martin-del-Rio 
in Spain (201 I). So far the results have paralleled those of K-12 
teachers in the US in terms of faculty adoption of new procedures. 

Expectancy for success. A concept related to self-efficacy 
was proposed by Wigfield and Eccles (2000), who included 
expectancy for success as one of the two main bases for motivation 
in expectancy-value theory, the other being value of the outcome. 
More specifically, this theory highlighted the subjective expectancy 
of an individual to achieve success at a task.The effects on behavior 
were very similar to self-efficacy. 

Need for competence. A third theory relating the impact 
of ability beliefs on motivation is Self-Determination Theory (SDT). 
Self-Determination Theory as proposed by Deci and Ryan (2000) 
posited that universal needs for feelings of competence, autonomy, 
and relatedness influence optimal functioning. Deci and Ryan stated 
that the extent to which these needs were satisfied was the critical 
element of self-determined motivation and willingness to take on 
new challenges. 

The need in SDT closest to self-efficacy was the need for 
feeling competent. The effect of this competence need would 
be connected to an individual’s perceptions of possessing the 
skills necessary to use an innovative practice such as data-based 
instructional improvement. This need was not identical to self¬ 


efficacy. Self-efficacy is a cognitive evaluation of potential success 
at a future task as opposed to a pre-existing need for feelings 
of competence (Pajares, 1996). Nevertheless both perspectives 
pointed to the belief in a faculty member’s own ability to succeed 
as a source of willingness to experiment with new ways to use 
student data to inform instructional improvement. 

Factor 2: Faculty Beliefs about theValue of Student 
Data for Improvement 

Utility value of data. Utility value refers to the faculty member’s 
beliefs about the ability of student data to inform instructional 
improvement. For example, Foley (201 I) explored K-3 teachers’ 
instructional behaviors in implementing a certain strategy. The 
choices they made were often tied to the usefulness the individual 
saw in a strategy. 

Expectations of desirable outcome. The expectations 
and values of an action were also part of theories from social 
psychology: the Theory of Reasoned Action (Ajzen, 1985) and 
its successor the Theory of Planned Behavior (Madden, Ellen & 
Ajzen, 1992). The Theory of Reasoned Action (Ajzen & Fishbein, 
1980; Fishbein & Ajzen, 1975) proposed that behaviors were the 
result of intentions, which arose from beliefs about the likelihood 
that a behavior would result in a desired outcome. These beliefs 
evolved from attitudes about the behavior and subjective norms 
(the societal or group standards) about the value of the behavior. 
These attitudes were based in part on the expected outcomes 
of performing the behavior, much like the value component of 
Expectancy Value Theory discussed earlier. Positive outcomes 
would lead to positive attitudes and greater tendency to perform 
the behavior. 

Value of social norms. Values are also a function of social 
pressures of the individual’s social network. If the behavior was 
socially desirable, the individual was more likely to engage in it. 
One could also tie this part of the theory to the value component 
of Expectancy Value Theory. In the current study we imagined that 
instructors might adopt innovative uses of student data if they 
believed doing so would lead to more efficient learning, and if other 
faculty were supportive of that outcome. 

Value of personal control. Madden, Ellen and Ajzen (1992) 
refined the Reasoned Action Theory by adding perception of 
individual control as a factor that influences choices. This theory 
was called theTheory of Planned Behavior. The difference between 
these two versions was the addition of the individual’s perceived 
control as a variable. The theory had two assumptions about 
direct influences: First, an individual, given sufficient information 
and resources, would pull together the positives and negatives of 
each action and make a rational choice. Second, once the individual 
had made the choice and intended to engage in the behavior, social 
pressures (both positive and negative) would affect whether or not 
the intention would be carried out. At this point the third variable, 
perception of personal control, became a factor in determining 
actions. The individual might make a good choice, but then believe 
that situational factors would work against a positive outcome 
(Fishbein & Ajzen, 2010). Behavior would be executed only in 
settings where high personal control was perceived.There is some 
question about whether perceived personal control is related 
more to decisions about self-efficacy or feasibility (Luszcrynska 
& Schwarzer, 2005). These two interpretations, self-efficacy (“I 
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will succeed at this”) and personal control (“I have control over 
the situation”) have been raised in the literature (Pajares, 1996). 
When the idea of being able to succeed at a task (self-efficacy) is 
contrasted with being in a situation over which one has control 
over completing a task (personal control), it is a fine line that 
separates self-efficacy and feasibility. 

Technology AcceptanceTheory. Calls for use of 
technology in education resulted in several more focused theories 
about diffusion of technology specifically. One theory, created 
by Hubona and Kennick (1996), was the Technology Acceptance 
Model. This model proposed that for acceptance a technological 
innovation had to be consistent with teacher values and beliefs 
about learning, be both useful and easy to use, and it had to inspire 
teacher confidence. These echo the initial factors of self-efficacy 
and value used in the current study. 

Van Acker, van Buuren, Kreijns, andVermeulen (2013) used the 
integrative model of behavior prediction from Fishbein (2000) to 
determine what variables influenced teacher adoption of digital 
instructional materials. Research based on this model indicated that 
attitudes and self-efficacy were the best predictors of teacher use 
of a new digital resource. 

In the present study, the Theory of Planned Behavior (Madden, 
Ellen andAjzen, 1992) would predict that an instructor’s intentions 
to collect and use student data would be influenced both by 
whether resources needed to accomplish the goal were available 
and if personal control over the class situation was present. Use 
would also be affected by colleagues’ opinions about whether they 
would use it in their own teaching. 

Factor 3: Faculty Beliefs about Feasibility of 
Collecting and Using Student Data to Improve 
Instruction 

The foregoing influential theories of social psychology also fit well 
with the next component of the Factors model in the present 
study - the factor of feasibility of implementation. We define this as 
the probability that a given task will be possible to complete, given 
the situation in which is carried out. In this study we broke this 
construct into more discrete units as described next. 

Personal control (Agency). Ryan and Deci (2000) 
proposed that feelings of autonomy were necessary for intrinsic 
motivation. In addition to believing student data were useful, an 
instructor must also believe that he or she had control over the 
decisions about environment and resources (personal control - 
has agency) or the conditions made it possible to engage in the 
task (context control). These ideas are related to two theories 
described earlier - the Theory of Planned Behavior (Madden, Ellen 
& Ajzen, 1992), which asserts that norms of the context and the 
perceived control by the individual influence implementation, and 
Self Determination Theory (Deci & Ryan, 2000), which points 
to feelings of autonomy (another way to characterize personal 
control) as key to motivation. Autonomy paired with perceived 
personal knowledge and capabilities produce greater intrinsic 
motivation (Ryan & Deci, 2000) and willingness to act. 

Context control. Contextual factors such as opportunities 
for choice and self-direction increase feelings of autonomy but also 
affect perceptions of do-ability. This meant that a belief that the 
environment made it possible to enact the intentions to innovate 
would influence a behavior choice. 


Interestingly, Andrade (201 I) discussed how pressures to 
use data for institutional purposes in accreditation could actually 
decrease faculty’s perceived autonomy for using student data 
in innovative ways. One way to avoid this would be to support 
faculty autonomy by encouraging them to become involved 
in designing assessment relevant to classroom goals. Andrade 
suggested administrative strategies such as increasing perceptions 
of administrative support through actions, and access to resources, 
could boost faculty feelings of autonomy and increase their 
motivation to use innovative practices. On the other hand, if 
policies or practices for collecting and using data run counter 
to a faculty member’s feelings of autonomy, he or she may balk 
at becoming involved in institutional level analytics, which would 
defeat the purpose of creating the analytics. Faculty input into the 
process must be solicited in a way to increase feelings of autonomy. 

Similar theories focusing on feasibility issues are found in the 
literature on technology acceptance. Mumtaz (2000) conducted 
a literature review of innovation acceptance and concluded that 
opinions of the teacher were critical to adoption. Mumtaz listed 
teacher beliefs about access to resources, quality of support, as well 
as ease of use as important to a teacher’s decision to innovate. In 
fact most of the literature on the spread of technology in education 
points to these same factors when it comes to integration of 
technology into the classroom. 

Adoption and Diffusion of Innovation. Perhaps the most 
well known theories that discuss contexts for innovation came 
from the business, healthcare sectors, and technological fields, 
for example, Rogers (2003) adoption and diffusion of innovation 
theory. In an early version, Rogers (2001) listed four main elements 
of diffusion of innovations. An innovation would diffuse due to 
I) characteristics of the innovation itself; 2) the communication 
channels available to make others aware of the innovation; 3) time; 
and 4) the social system into which it was diffusing. 

The characteristics of the innovation that facilitate its adoption 
included its relative advantages over the existing system; its 
compatibility with the beliefs and values of the potential users; how 
difficult it was to understand;“trialability” or the opportunity to try 
it out first; and observability - the degree to which others can see 
it work. In terms of communication channels, Rogers concluded 
it was the personal communication channel between peers that 
seemed to have the biggest effect on adoption and diffusion. 

A recent attempt to use the diffusion model to understand 
problems in innovations in engineering education (Borrego, Froyd, 
& Hall, 2010) allowed us to see how contextual factors seemed to 
overwhelm those trying educational innovations.The authors were 
tracking the acceptance of seven different instructional innovations 
for engineering education. They reported that although 82% of 
the department chairs were aware of the innovations, only 47% 
reported having adopted the innovations to some extent in their 
departments. Over half of the comments about barriers cited 
resource limitations as the biggest cause of failure to innovate. 
The second largest category was faculty issues. Borrego, Froyd 
and Hall said “department chairs stressed that adoption of 
educational innovations is heavily reliant on participation of faculty 
members.” (pg. 199) They continued by citing “Faculty time for 
preparation and management of labor intensive innovations,... 
the culture of engineering higher education,... faculty resistance 
to change, marginalization of teaching in promotion and tenure, 


and skepticism regarding evidence of improved student learning.” (pg. 
199) All of these can be seen in the discussions of context factors 
that impede diffusion of innovation and factors that can keep faculty 
from experimenting with student data and innovations in instruction. 
In another insightful research on diffusion, Macfadyen and Dawson 
(2012) found that those making recommendations for changes were 
“assessing the degree to which any change will burden themselves 
and their colleagues with the need to learn how to use complex new 
tools, and/or the need to redesign change their teaching habits and 
practices, without offering any appreciable advantage or reward.”(pg. 
160) The feasibility factor can take many shapes, but convenience and 
low effort appear in many guises to affect innovation. 

Integrating the Factors to Encourage Faculty Use of 
Data 

Drawing on common elements from the literature, the current study 
analyzed how faculty perceptions of self-efficacy for collecting and 
using student data, perceived value of student data for helping to 
improve instruction, and their agency and the feasibility for being able 
to use student data were related to their actual data collection and 
use. Data collected from the faculty in the current study followed 
the Factors model shown in Figure I. Here self-efficacy, value, and 
feasibility (personal and conditions) were proposed as the major 
factors in faculty decisions to collect and use student data to improve 
instruction. The following research questions were addressed: 

Research Question I 

Factor I: Faculty Self-efficacy for Collecting and Using 
Student Data 

I A. How high did the faculty in the sample rate their self- 
efficacy for collecting and using student data? 

I B. What was the correlation between faculty reported self- 
efficacy for collecting and using student data and their use of 
a reflective student data-based improvement process? 

Research Question 2 

Factor 2: Faculty Beliefs about the Value of Student 
Data for Instructional Improvement 

2A. How much did the faculty in the sample value student 
data in terms of its usefulness for instructional decisions? 

2B. What was the correlation between faculty reported 
value of student data and their reported use of the reflective 
student data-based improvement process? 

Research Question 3 

Factor 3: Faculty Beliefs about Feasibility of Collecting 
and Using Student Data to Improve Instruction 

3A. How strongly did the faculty in the sample believe that 
it was feasible for them to collect and use student data for 
instructional decisions? 

3B. What was the correlation between faculty beliefs 
about the feasibility of collecting and using student data 
and their reported use of the reflective student data-based 
improvement process? 

Research Question 4 

Development of measurable outcomes of student data 
use 


4A. To what extent did the faculty in the sample report the 
collection and use of student data in the past? Were some 
kinds of data collected more frequently than others? 

4B. To what extent did the faculty use the reflective 
processes involved in the reflective student data-based 
improvement process? 

Research Question 5 

Relationships between model factors and outcome 
variables 

5A.What did regression of data types used on self-efficacy, 
value, and feasibility show about the strength of any effect of 
any of the studied variables? 

5B. What did regression of the reflection processes on 
number of types of data used, self-efficacy, value and 
feasibility show about the relative strength in affecting the 
target variable? 

The Present Study 

We have drawn on the above theories to inform our investigation 
of faculty use of student data. For the quantitative investigation of 
our hypotheses we predicted that participants who had high scores 
on measures of self-efficacy, data value, and feasibility of collection 
and use would also show a higher level of use of student data in the 
past and more use of the reflective processes of data use to improve 
instruction. Qualitative methods based on intensive interviews with 
the participants were used to add to and verify our predictions using 
their own words. 

METHOD 

This study consisted of both quantitative and qualitative data gathered 
from faculty representing a range of disciplines across a large 
southwestern university. Data were collected during the spring and 
fall semesters of 201 1-12 and represent faculty perceptions and use 
of student data prior to the onset of a new teaching initiative at the 
institution. 

Participants 

Forty-one faculty participated in this study. (Because not everyone 
participated in both quantitative and qualitative parts of the study, 
occasional discrepancies in total responses occur.) Demographics of 
the participants are shown in Table I. Procedures for protection of 
the participants and confidentiality of their information were guided 
by IRB human subject requirements of the University. Faculty who 
participated in this initial data collection were instructors in large 
undergraduate classes that were targeted for redesign (n=21) plus 
faculty who were matched to the instructors in terms of rank, gender, 
and college (n=20) and agreed to respond to the survey component 


TABLE 1. Sample Demographics 

College 

Liberal Arts 

Natural Science 

Professional 



21 

12 

8 


Rank 

Lecturer 

Assistant Prof 

Associate 

Prof 

Full Prof 


19 

s 

13 

3 

Years Teaching 

l-S 

6-10 

1 1-20 

>20 


S 

7 

15 

13 
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with a follow-up interview as well. 

Institutional Information 

The institution at which the study was conducted is classified by 
the Carnegie Classification 2015 version as a Doctoral University: 
Highest Research Activity. There are approximately 64,000 students 
and 3090 faculty in 18 school and colleges. This data collection was 
a part of a campus wide initiative to improve the instruction in large 
undergraduate courses. 

Measures 

The data gathered by the online surveys consisted of the following 
quantitative sources. 

Data related to past use of student data. The following 
two variables were benchmarks representing patterns of data use by 
faculty before the start of the project. 

Outcome measure I: Prior use of student data. The prior use 
survey asked faculty to check any of six types of student data they had 
used in the past, including an option to indicate that the individual did 
not use student data to modify instruction, and an option to suggest 
other types. The purpose of these items was to establish a baseline 
of types of data used by these faculty. The types of student data 
were selected as the most commonly used (See Table 3.) They were 
compiled from suggestions of two experienced faculty developers, 
each with at least 30 years of working with faculty, and worded to 
focus on the instructional improvement goals of assessment. Items 
were worded generally and an example of each was given in order to 
be recognizable to the widest range of disciplines. 

Outcome measure 2: Frequency of engaging in the 
reflective student data-based improvement process, (adapted 
from College Teacher Sense of Self-efficacy (CTSES), Prieto Nevarro, 
2005). The survey on use of the reflective process asked how 
often the respondent engaged in nine reflective activities used for 
gathering and interpreting student data (e.g. “In your teaching, how 
often do you design data collection strategies for monitoring what is 
happening in class?”) and making instructional improvements based 
on data in the reflective student data-based improvement process 
(e.g. “In your teaching, how often do you reflect on your teaching 
practices with the aim of making appropriate improvements?”). The 
survey used a six-point scale from I - never to 6 - always. Items 
representing components of the reflective process can be found in 
Table 5. Cronbach’s alpha on this scale was 0.83. Slightly reworded 
item stems from this measure were also used for Factor I - Self- 
efficacy for gathering and using data (see below). 

Data related to the current model. The following data 
focused on aspects of our proposed theoretical model including (I) 
self-efficacy for the gathering and analyzing student data, (2) value 
of student data for changing instructional practice, (3) feasibility of 
gathering student data, and (4) effort needed to gather and analyze 
data relative to other teaching responsibilities. 

Factor I: Self-efficacy for gathering and interpreting 
data and making improvement based on the data (adapted 
from CTSES, Prieto Navarro, 2005).This self-efficacy survey consisted 
of nine statements in two sets - five representing self-efficacy for 
gathering and interpreting data (e.g. “I am confident that I can use 
student data to design data collection strategies for monitoring 
what is happening in class?”) and four representing self-efficacy for 


making improvements to teaching based on data (e.g.“I am confident 
that I can use student data to interpret student learning in a way to 
plan instruction.”). Participants rated statements from I - strongly 
disagree to 6 - strongly agree. Items are shown in Table 4. Note that 
the statements parallel those from outcome measure 2 described 
earlier. The Cronbach alpha for the overall self-efficacy scale was 
acceptable at .83. 

Factor 2: Confidence in the value of student data. This 
value survey asked the participants to rate their confidence that 
student data use could support various instructional tasks. Participants 
rated nine statements from I - strongly disagree to 6 - strongly agree. 
For example, an item asked faculty to rate their level of agreement 
with the statement “I am confident that using student data will make 
a difference in the effectiveness of my course.” The Cronbach’s alpha 
for this scale in this sample was acceptable at .88. 

Factor 3: Feasibility of using student data (developed for 
this study). This feasibility survey assessed participants’ confidence 
that they had the authority, flexibility, resources, and support of others 
to use student data to modify instruction (seeTable 6). The four items 
were rated on a six-point scale ranging from I - strongly disagree 
to 6 - strongly agree. For example, faculty rated their agreement 
with the statement “I am confident that I have the authority to use 
student data to make decisions about instruction in the course.” The 
Cronbach’s alpha for this scale was acceptable at .73. 

In the Factors model but not included in this phase: 
Effort of using student data (developed for study). Effort in this 
context refers to amount of time and attention that must be put forth 
in order to engage in a task. At this point most faculty did not have 
enough experience with student data use to make a reliable estimate 
of the time required. Therefore, these data were not included in the 
analyses. 

Procedures for the Quantitative Part of the Study 

Data were collected during the fall and spring semesters of 201 I - 
12. Participants received an e-mail invitation to participate, including 
a recruitment statement, a consent to participate document, and a 
model of the overall plan of research. If faculty chose to participate, 
they would click on the link to the survey to begin responding. This 
response also documented their consent to participate. 

Because this study was part of a new teaching initiative aimed 
to redesign large lecture-oriented courses at the university, part of 
the evaluation procedures required a baseline understanding of how 
(and if) faculty used information about their students to inform or 
influence their teaching practice and course design. Participants first 
responded to online surveys (described above under “measures”) 
administered through Qualtrics regarding the components of the 
Factors model that were of interest: prior use of student data, prior 
engagement in reflective instruction improvement, self-efficacy for 
gathering and using data, value of data, and feasibility of using data to 
improve instruction. Following the completion of the survey, faculty 
were asked if they would consent to an interview to provide more 
in-depth information to their survey responses. 

QUANTITATIVE RESULTS 

Descriptive Statistics of Survey Data 

Means and standard deviations for the main variables are provided in 
Table 2 for summary purposes. Each variable is discussed separately 


TABLE 2. Means and Standard Deviations for Main 
Variables ofTotal Sample. 


Variable 

Mean (sd) N=4I 

Prior use - Previous use of different types of 
student data (# per person) 

3.46 (1.61) 

Number of different types 
used 

Self-efficacy - instructors’ belief in their 
ability to gather and interpret data and make 
improvement based on the data (scale 1-6) 

4.66 (0.62) 

level of confidence - 

higher equals higher level 

Value of student data - confidence that 
student data use could support various 
instructional tasks (scale 1-6) 

4.42 (0.65) 

value placed on data - 

higher equals more value 

Feasibility - instructors’ belief that a given task 
will be possible to complete, given the situation 
in which it is carried out (scale 1-6) 

4.67 (.77) 

feasibility of using data 
- higher equals higher 
feasibility 

Frequency of use of the reflective student 
data-based improvement process. - refers 
to the instructor’s use of any of 9 strategies of 
careful gathering and analysis of the data shown 
in Table 5. (scale 1-6) 

3.84 (.82) 

frequency of use - higher 
equals more use 


below. 


Factor I: Instructor self-efficacy for using student 
data to reflect on and improve instruction. 

Question I A - Level of self-efficacy. To answer research question 
I A, we used the adapted CTSES to examine reported self-efficacy for 
using data. Figure 2 shows the percent of participants reporting self- 
efficacy in either gathering data or using it for improvement. Eighty- 
seven percent of faculty responded that they felt confident in their 
ability to gather student data. Ninety-four percent reported that 
they were confident in their ability to use the data they collected to 
improve instruction.The average level of overall self-efficacy for data 
collection and use was 4.67 (.61) with higher means associated with 
higher self-efficacy. 

Question IB - Relation to use of the reflective process. 

Exploring further, we found that the correlation between self-efficacy 
for student data collection and use and the actual use of the reflective 
student data-based improvement process was 0.75 (p =.001). Those 
who were confident in their ability to use student data were also 
likely to report engaging in the reflective process for data use. We 
will see later that while the correlation with actual use is high, the 
percent of faculty reporting that they actually used the process was 
lower, specifically, 39% for gathering data but also 73.75 % for using 
the data to improve instruction (Figure 3). 

Factor 2: Instructor beliefs about the value of data. 
Question 2A -Value of data. One source of failure to engage in 
data gathering could have been a belief that such data are not useful. 
To address research question 2A, we examined instructor ratings of 
the value of student data. The overall mean of the value scale was 
4.42 (0.65) on a six point scale with the higher end representing “very 
confident" in the utility of student data for instructional improvement. 
Figure 4 summarizes faculty confidence in value of student data 
for each of the listed instructional tasks. Each bar represents a 
potential contribution of data use (e.g. increasing the effectiveness 
of instruction). Over 80% of participants agreed with the usefulness 
of data in most areas. The one lower item (75%) was the possibility 


TABLE 3. Percentage of Faculty Reporting Use of Each 
Type of Data 


Type of data use 

Percent of respondents 
reporting this use 
(N=4I) 

Before the semester to get an idea of who 
would be in the class (for example, looking 
at the class rosters for student levels and 
majors). 

55% 

At the beginning of the course to measure 
student prior knowledge (for example, doing 
an early baseline quiz or survey to see what 
the students seem to already know). 

30% 

At the beginning of the course to measure 
student motivation and interest (for example, 
doing a survey on the first day of class to see 
what goals students have in the class). 

20% 

During the course but outside the context of 
an exam to measure student understanding 
for the purposes of instructional changes at 
that moment (for example, asking questions 
or using clicker surveys to get immediate 
feedback on student understanding in the 
moment). 

52.5% 

After an exam, using the exam results to 
modify instruction (for example, identifying 
concepts that seemed to be difficult for 
everyone and might need review). 

75 % 

At the end of the semester, using student 
performance or evaluation to modify future 
classes (for example, gathering student 
comments about what helped and hindered 
their learning overall). 

82 . 5 % 

1 do not regularly use student data to modify 
instruction. 

12.5% 


of student data use in raising student course evaluations. From these 
results it appeared that faculty believe student data could be useful 
in many ways. 

Question 2B - relation to use of the reflective process. 

The correlation between believing in the value of student data and 
the actual use of the reflective student data-based improvement 
process was 0.63 (p =.001). Those who saw value of student data 
also reported engaging in the reflective process for data use. 

Factor 3: Instructor beliefs about the feasibility of 
gathering and using student data. 

Question 3A - Feasibility of collecting and using data. To 

address research question 3A, we asked participants to rate the 
feasibility of student data gathering and use. As was seen in Table 3, 
participants reported higher use of data that were being gathered 
for other purposes or by other parts of the institution (enrollment 
information, end of course surveys, or exams). Faculty may be 
influenced not by how much they value data, but how difficult it 
is to collect and use. We have labeled this factor “feasibility” and 
identified four aspects: authority to make a change, flexibility to 
change, resources to support the change, and peer and administrative 
support. Over 87% of faculty felt that they had the authority, flexibility, 
and administrative support to improve instruction with student data. 
(See Figure 5.) In contrast, only 70% of faculty reported having the 
resources to gather data for instructional improvement. The overall 
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TABLE 4. Self-Efficacy for Use of Student Data for Re¬ 
flecting on and Changing Instruction 


Please select the level best reflecting your 
agreement with the statement. 

You are confident that you can: 

1 = not at all confident 

to 

6 = very confident 

1. Reflect on your teaching practices with the aim 
of making appropriate improvements. 

5.30 (.68) 

2. Design data collection strategies for 
monitoring what is happening in class. 

4.13 (.95) 

3. Use different evaluation methods to assess 
student learning. 

4.58 (1.026) 

4. Interpret student learning data in a way to plan 
instruction. 

4.50 (.81) 

5. Adapt teaching practices in response to your 
students’ evaluations of your teaching. 

4.59 (1.12) 

6. Decide on the most appropriate evaluation 
method for a particular course. 

4.73 (.81) 

7. Employ systematic methods that permit you to 
assess your own teaching. 

4.20 (.99) 

8. Adapt to the needs of your students when 
planning class sessions and activities. 

4.95 (.95) 

9. Be flexible in your teaching strategies even if 
you must alter your plans. 

5.03 (1.07) 

Overall Mean 

4.67 (.71) 


mean of the feasibility scale and means for each of the four items 
are shown in Table 6. In comparison to the other factors, overall 
feasibility is on a par with self-efficacy at a mean of 4.67, but its lowest 
rated component, resources, may be holding it back. 

Question 3B - relation to use of the reflective process. 
The correlation between feasibility of gathering student data and the 
actual use of the reflective student data-based improvement process 
was 0.55 (p = .001). Those who believed that it was possible to 
gather and use student data were also likely to report engaging in the 
reflective process. 

Outcome measures of student data use 
Question 4A - Baseline measure of prior data use. To answer 
research question 4A, we examined past use of various types of 
student data. Table 3 provides the percentage of faculty reporting 
they had employed each of the data listed. Twelve and one half 
percent of the sample indicated that they did not use student data. 
The average number of different types of data used by the remaining 
faculty members was 3.46 types with a range of 0 to 6 types and 
a standard deviation of 1.61. Over 50% of the participants used 
end-of-class surveys, exam results, student demographics, and in 
class assessment to modify their instruction. Less than 30% of the 
participants gathered information about students’ prior knowledge 
or motivation and interest of students in order to guide instructional 
plans. Half of the participants used student information prior to the 
semester to understand the make-up of the class. Note that the 
highest levels of use, which were end-of-semester surveys to improve 
future classes, actual exam results used to plan remediation (both 
indicated in bold in Table 3), and student information prior to the 
semester, are collected for other purposes or provided by other 
parts of the institution - that is, they are easy to collect and available 
without extra effort. 

Question 4B - Use of reflective student data-based 


TABLE 5. Actual Use of Student Data for Reflecting on 
and Changing Instruction 


In the following items, please choose the 
responses that best fit your situation. 

Mean (sd) 

1 = Never 

to 

6 = Always 

In your teaching how often do you: 

1. Reflect on your teaching practices with the aim 
of making appropriate improvements? 

4.83 (1.05) 

2. Design data collection strategies for 
monitoring what is happening in class? 

3.07 (1.03) 

3. Use different evaluation methods to assess 
student learning? 

3.71 (1.10) 

4. Interpret student learning data in a way to plan 
instruction? 

3.78 (l.l 1) 

5. Adapt teaching practices in response to your 
students’ evaluations of your teaching? 

3.95 (1.26) 

6. Decide on the most appropriate evaluation 
method for a particular course? 

3.71 (1.21) 

7. Employ systematic methods that permit you to 
assess your own teaching? 

3.00 (1.30) 

8. Adapt to the needs of your students when 
planning class sessions and activities? 

4.22 (1.19) 

9. Be flexible in your teaching strategies even if 
you must alter your plans? 

4.32 (1.19) 

Overall Mean 

3.86 (.82) 


improvement process. As noted earlier, Figure 3 shows that a 
higher percentage of participants report engaging in practices to 
improve instruction (73.75%) than engaging in systematic data 
gathering (39%), despite reporting self-efficacy in Figure 2.The average 
overall for actual use of the reflective processes was 3.86 (.82). Means 
for each of the items on this scale are shown in Table 5, and are lower 
than the reported mean of self-efficacy for the matching design and 
use item in Table 4. 

Predicting the use of student data 

Question 5A - Predicting prior number of uses of data. We 

attempted to identify the factors in Figure I that appeared to be 
related significantly to levels of actual use. To address this question, 
the number of different data types that a faculty member reported was 
regressed on self-efficacy, value of the data, feasibility, and frequency of 
actual use of the reflective student data-based improvement process. 
Of these variables, both feasibility (p= -.663, t=-2.123) and use of 
reflective processes (P= 1.550, t=3.884) were significant predictors 
of how many different types of data were used by faculty (p<.05). 
Note that feasibility is negatively related to the number of types of 
data used, suggesting that when faculty believe that there are many 
barriers to data use, they use fewer types. 

Question 5B - Predicting use of reflective process. 
Additionally, when treating use of reflective processes as the outcome 
variable, both number of types of data used (P=. 191, t=3.884) and 
self-efficacy for use (P=.6 13, t=4.202) were found to be significant 
predictors (p =.001). In other words, a faculty members’ confidence 
that he or she can gather and analyze student data was related to 
engagement in reflection on data use to improve instruction and the 
variety of data types used. 

Returning to our initial questions of whether high scores on the 
variables identified in the Factors model shown in Figure I would be 
associated with use of reflective processes, we found that self-efficacy 
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Figure 2. Percent of faculty reporting self-efficacy for gathering 
and using student data to improve instruction 

and feasibility were predictors of use of the reflection process and 
merit further examination. In these regressions, value of the data 
did not reach significance, although it was significantly correlated 
with using the reflective practice process. This finding is contrary to 
what is found in both the motivation literature and the innovation 
literature. The reasons for this difference need to be explored in 
greater depth. 

Summary of Quantitative Data 

The survey data showed that 50% or more of the faculty in this 
sample did use some student data for improvement, particularly 
those data that were being gathered on a regular basis for other 
purposes, such as exams and course evaluations by students. They 
also reported having confidence in their own ability to gather and 
use data, but fewer reported actually using the reflective student 
data-based improvement process activities.The reported self-efficacy 
appeared to be an acceptable predictor of faculty use of student data. 
Except in the case of data improving student evaluations, the faculty 
reported valuing data for use in many phases of instruction. As to 
the other variables, faculty reported having the authority to modify 
instruction based on data, the support of administration to do so, 
and the flexibility to modify their course. The one area where their 
confidence was not as high is whether or not they had the resources 
to help them gather and use data. 

Qualitative Component of the Study 

To complement our quantitative data and create a better 
understanding of how faculty perceive and use student data, the team 
collected qualitative data through interviews. The team interviewed 
faculty about their instructional use of data. The qualitative responses 
revealed themes supporting the quantitative findings. The following 
sections describe participants, coding procedures, approach for 
analysis, and results. 

Participants 

Interviews were conducted with 29 of the participating faculty who 
agreed to be interviewed. The interviews were audio-recorded, and 
sections relevant to our research questions were transcribed and 
coded. The coding process is described in the following section. 


TABLE 6. Faculty Perceptions of Feasibility to Use Data 
in their Situation 


Component of feasibility 

Mean (sd) 

6 point scale with higher values 
equaling more support 

Overall composite 

4.67 (0.77) 

Authority to make a change based on data 

5.07 (0.82) 

Flexibility to make a change based on data 

5.02 (0.88) 

Resources to support the change based 
on data 

4.02 (1.15) 

Peer and administrative support for using 
data to make a change 

4.56 (1.25) 


Coding process 

The team used a thematic coding approach (Coffey & Atkinson, 
1996). This approach allowed the theoretical Factors model to 
guide interview questions and provided opportunity to assess 
model accuracy in describing faculty attitudes. Additionally, the team 
employed the constant comparative method (Corbin & Strauss, 2008) 
to compare findings and develop a component chart that improved 
code validity and reliability. 

Peer-debriefing allowed the team to discuss problems and 
consider unpredicted findings. Standard inter-rater reliability methods 
were used to improve agreement through discussion. Inter-rater 
agreement across pairs of raters showed an average agreement 
level of 79.23%, acceptable for these data according to the Center 
for Educator Compensation Reform (Graham, Milanowski, & Miller, 
2012 ). 

Components. Initially four factors were used, representing 
each factor in the Factors model. They included self-efficacy (“Can 
I personally do this?”), value (“Is this worth doing?”), feasibility ( 
“Would I be allowed?”), and effort (“What would it take to do this?”). 
Although effort was present in the model, it was not included in the 
quantitative analysis as noted earlier. Since it was mentioned by some 
of the interviewees and therefore could have provided some insight 
into this factor separately effort was kept for qualitative analyses. 

The team added two factors by dividing the student data use 
component into actual use, stated as recollections of experiences with 
collecting and using data, and Intended use. The team noticed faculty 
expressed attitudes specifically related to data collection they planned 
to implement but had not enacted. This was not conceptualized in 
the Factors model, but these sentiments arose frequently enough that 
the team decided a distinct component was necessary. 

During final coding, the team finalized and used these six 
components: actual student data use, Intended student data use, self- 
efficacy, value of data, feasibility, and effort. 

QUALITATIVE RESULTS 

In the following section, findings from the qualitative data are 
described. Here, frequency of codes are discussed and excerpts are 
provided to support our interpretations. 

Actual Data Use 

Actual use referred specifically to student data already being used by 
faculty While many data types were mentioned, the most common 
were end-of-semester course evaluations, grades and accuracy rates 
from exams, and responses to iCIicker questions collected in class. 
The following is a good example of multiple ways a faculty uses 
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Figure 3. Percent of faculty reporting actual use of practice of gathering or using 
data to improve instruction 

student data. 

“I’m interested in exam data. But then it also helps me 
because when I get that data back I write into my final exam 
key copy what percentage of students got each question 
right. And so if I’m noticing that a lot of learning outcome 
l-l was missed, then I can say,‘okay I’m not doing a very good 
job teaching that’. Or if I see just a particular question that 
a lot of people missed I can say it is a very good question.” 

In general this finding supports the quantitative data finding where 
end-of-course evaluations and exam results were the most commonly 
reported data used. Clicker use was also frequently noted, but has no 
counterpart in the quantitative results. 

Some faculty reported using data infrequently and with less 
confidence. Some faculty seemed unsure of what was meant by 
“student data” and restricted their use to the most frequently 
encountered, such as exam scores. They also did not know what 
various data types were possible, or how to interpret and use data 
for improvement. 

Intended Data Use 

The concept of intended data use, not originally conceptualized 
in the model, was created during the coding process due to its 
frequent occurrence. Intended use referred specifically to plans faculty 
mentioned about future data use. 

“I would say the pre and post assessment, I’ve not done a 
good job with addressing any of that. We need to work and 
have better pre and post assessments.” 

Comparatively, this intention component typically occurred in 
interviews with faculty already using student data. 

Self-Efficacy 

Self-efficacy referred to perceptions of competence with data 
collection and use. Self-efficacy also categorized as high or low, and 
sometimes both categories were coded for one participant.This trend 
expands our quantitative self-efficacy findings because it seems these 
perceptions can be contextual. One high self-efficacy excerpt reads, 
“That’s one thing we’ve done well is we know when we 
want them all to nail it and we know when we want them 
to be confused and I think we’re getting very, very good at 
[writing clicker questions].” 

For other faculty, codes generally trended either high or low. An 
excerpt from a participant with low data self-efficacy reads, 



Figure 4. Percent of faculty reporting confidence in data to inform various aspects 
of teaching a course, a measure of the value of student data. 


“I’m not very good at it, so when I sit in meetings and they 
have a bunch of spreadsheets I didn’t create, I don’t know 
what it’s all telling me. So I let other people tell me what 
it’s telling me.” 

Value 

The value component was the most frequently coded along with 
actual data use. Most of the value codes were positive. Faculty 
usually mentioned perceptions of high personal value for the data, 
but sometimes would also discuss value students placed on data use. 
Further, several faculty noted the potential student data use has for 
positively impacting learning. The following are both positive value 
excerpts from two different faculty: 

“Clicker questions are very, very good. And the students 
like it. It’s a very engaged class. They’re all clicking, and if 
everybody does well they cheer.” 

“But a lot of times because of that information I will change 
the rest of the semester. Usually the students like that I 
pause, and I see they have questions and spend a lot of time 
doing that.” 

Feasibility 

Feasibility was the least frequently coded. Feasibility referred to 
institutional resources and support related to data collection and use, 
addressing perceptions of authority to access or interpret data.When 
noted, it was generally in a negative context. For example: 

“The demographics and all that, I don’t know if we have 
much access to some of that material.” 

In general, faculty reflections on institutional support resources 
showed negative perceptions or just lack of awareness.The following 
excerpt shows one exception. 

“We have a coding team that works for CNS (College of 
Natural Sciences). So I say I need problems on absolute 
values and then they generate some and I put them in my 
work. So there’s a big bank in this computer system of 
homework problems they can pick from.” 

Effort 

Effort was also coded with moderate frequency. Effort referred 
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Figure 5. Percent of faculty reporting a belief that they had authority, flexibility, 
resources, and support to make changes in their teaching based on student data. 

to perceptions of resources and expenditures required for data 
processes, Most codes illustrated facultys’ perceptions of high 
amounts of effort needed to collect, interpret, and use data. Typically 
these perceptions referred to large classes. High effort perceptions 
sometimes deterred faculty from collecting student data as shown in 
this excerpt. 

“Sometimes I’ll do the minute thing....And it’s hard to do 
with 200 in a large class. So I don’t do that so often. I try to 
do that more with my smaller classes." 

Despite the effort required, some faculty collected data despite high 
effort perceptions, and using other resources made this easier. This 
excerpt shows one participant utilizing assistants to collect essay data. 
“I mean I grade probably 25 tests.The TAs do much of the 
grading so I get feedback from them.” 

These trends supported the quantitative findings because the most 
frequently used data involved low collection efforts since they were 
already collected by the university (e.g. course evaluations and exam 
data) or could be collected electronically (clicker questions). 

GENERAL DISCUSSION 

Interweaving the Quantitative and Qualitative 
Findings 

Factor one: Self-efficacy to collect and use data for 
improvement. 

Quantitative results. Faculty in this sample rated their knowledge 
and ability in instructional improvement at a fairly high level overall 
as seen in Table 2. Comparing Table 4 (self-efficacy) with Table 5 
(use of the process) we see that self-efficacy did not translate into 
use of improvement practices, as shown by the lower means on 
the comparable use items. On the other hand, the high correlation 
between self-efficacy and the use of ways to reflect on improvement 
(r = 0.75, p=.00l) indicated that those who are confident are also 
more likely to report use of the data process to improve. We believe 
self-efficacy is a circle; the more confident one is, the more one is 
willing to try, and the more one tries successfully, the more self- 
efficacy is developed. 

Qualitative results. In terms of high self-efficacy being an 
important predictor of success, the interviewees mentioned this 


factor with moderate frequency in comparison to other factors. An 
interesting nuanced interpretation of self-efficacy that the interviews 
raised was the fact that self-efficacy can be high OR low and have 
different impacts on the individual’s behavior. Low self-efficacy might 
not be on a continuum with high self-efficacy, but rather orthogonal, 
resulting in a different set of unique beliefs, attitudes, and behavior. 
Although continuous levels were implied in the scales used, the 
possibility of orthogonal continua was more obvious by the faculty 
comments during interviews. 

Factor two: Value of student data. 

Quantitative results. Value of the data was evident in survey 
responses. The overall mean on the value items was 4.42 (0.65) 
in Table 2. While not the highest main factor mean, it is above the 
middle of the scale, indicating that faculty had a positive impression 
of student data use for improvement. There was also a positive 
correlation between an instructor’s valuing of data and use of the 
student data-based reflection process (r = .63, p = 001). As with 
self-efficacy, instructors who believed student data could be useful 
in instructional improvement were also likely to report using the 
reflective data-based methods. Faculty may be ready for more 
sophisticated uses of data at this point. 

One less obvious phenomenon with regard to the value of 
student data was that the actual number of different types of data 
used was not very diverse. The alternatives being used were ones 
that didn’t require much initiative on the part of the instructor.Those 
data were collected for a different use, usually on a fairly regular 
schedule by others. While these are useful data, they do not capture 
the full range of student learning and therefore may not uncover real 
problems causing poor performance. 

Qualitative results. The value of student data was the most 
frequently mentioned comment made in the faculty interviews. This 
supported the quantitative findings of high value placed on student 
data. All the comments about student data spoke to its positive value. 

Here, too, there was a more nuanced interpretation than was 
present in the quantitative data. Comments made by faculty also 
indicated a recognition that the students benefited from the collection 
of their data, helping them recognize their progress, successes and 
failures. Perhaps the multiple recipients of value (like students) need 
to be considered when measuring overall data value. 

Factor three: Feasibility. 

Quantitative results. Overall results of the survey items 
assessing feasibility had a relatively high mean of 4.67 (.77) in Table 2. 
This would indicate that faculty believed it was feasible to collect and 
use student data for improvement. In addition, the overall feasibility 
score was positively correlated with use of the reflective data-based 
improvement process (r = .55, p = 001). Of the four subcomponents 
of feasibility, availability of resources was the lowest, indicating that if 
there was something amiss with feasibility, it was whether the faculty 
had the resources to go forward. 

Qualitative results. Feasibility was not a factor mentioned 
by faculty spontaneously, but when it was, the responses tended 
to highlight the lack of resources. This observation supported the 
quantitative results with regard to perceptions of the lack of resources 
noted in that item of the survey. 
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Overall Support for the Factor Model 

Our purpose for this study was to evaluate the proposed model 
for factors affecting faculty use of student data for instructional 
improvement. We have found that the three factors, self-efficacy, 
value of data, and feasibility, suggested by the literature and included 
in this model have a legitimate claim to being able to influence faculty 
use of student data in making instructional improvements. The results 
suggest that paying attention to these factors could encourage faculty 
to be more systematic and productive in their use of student data. 

LIMITATIONS OF THIS STUDY 

In any research study, there were limitations affecting our ability to 
make definitive statements about connections between the data 
collected. We list them here and their potential impact plus any 
solutions that we have considered. 

Termination of project before completion. 

The biggest impact on our ability to draw causal conclusions was 
caused by the project being terminated before the intervention and 
post measures could be taken. We were able to gather most of the 
pre-intervention data, dealing with pre-existing faculty attitudes and 
beliefs about student data and past data sources they had employed. 
The unavailability of post-intervention data limited what we could say 
about changes in faculty beliefs and attitudes when given additional 
support and resources. 

Faculty self-report as sole data source. 

As in most faculty development studies, the data were based on self- 
reports by the faculty. Cross-checking between the quantitative and 
qualitative data helped to show that the responses were relatively 
consistent across measurement modes. 

It is a concern of research in post-secondary settings that there 
are not better ways to measure the key constructs. Observational 
data would have been a good benchmark to test the veracity of faculty 
self-reports. Since attitudes and beliefs will probably always include 
qualitative methods, strategies for gathering data from college level 
faculty might need to be focused and standardized to increase the 
replicability (and therefore the respectability) of the data. We also 
suggest that research in SOTL converge on a set of more standard 
quantitative instruments to allow data to be compared more readily. 

Small sample size. 

The present study had a small number of participants (41). This 
limited the ability to generalize from these data to the large post¬ 
secondary population of faculty. The study should be repeated as 
originally planned. 

Creating a More Generally Supported Model for 
Faculty Use of Student Data. 

The overall theoretical model underlying this study was social 
cognitive theory as applied to choices. This is currently the most 
widely used model of behavior change (Luszczynska & Schwarzer, 
2005). The primary premise of social cognitive theory is that in 
making choices about behaviors, an individual’s cognitions act as a 
mediator between what is happening and the responses that the 
individual makes. As a result the same situation can be viewed entirely 
differently based on interpretations each individual makes in the 


moment. Choices are more a function of the individual chooser than 
the objective reality of the situation. 

Social cognitive theory has been applied in a wide variety 
of circumstances where individuals are making choices, in health 
behaviors, in technology use, and many others. In the present study 
we were looking at the key factors drawn from social cognitive theory 
to understand why faculty would or would not choose to use student 
data in ways to improve the learning and student success. 

A particularly useful discussion of the many faces of social 
cognitive theory was provided by Connor and Norman (2005b), 
from a workshop sponsored by the National Institute of Mental 
Health on promoting HIV-preventive behaviors. Individuals in 
attendance included many of the major theorists who worked within 
the framework of social cognitive theory. Connor and Norman 
reported that the experts “...identified eight variables which, they 
argued, should account for most of the variance in any (deliberative) 
behavior.” (pg. 18). These were (slightly modified for length and clarity 
here): I) a strong intention (or motivation) to perform a behavior; 
2) the necessary skills to perform the behavior; 3) an absence of 
constraints on the behavior; 4) a cost benefit ratio in favor of the 
behavior; 5) more social pressure to perform than not to perform 
the behavior; 6) a behavior consistent with the individual’s self-image; 
7) no expectations of the outcome to be negative emotionally; and 8) 
high levels of self-efficacy (pg. 19-20). Connor and Norman referred 
to this as “the ‘major theorists’ model" (pg. 20). 

This major theorists’ model is very similar (though more 
inclusive) to the proposed factors used in this study. Most of the 
factors involved interpretation and rational decisions about whether 
to perform the behavior in light of its feasibility (in our case, to collect 
and use student data to improve instruction). We have envisioned 
them (in a different order from Connor and Norman) as: 

1. The faculty member must have self-efficacy for data 
collection, interpretation and use for improvement. 

2. The faculty member must value the potential contributions 
that student data can make to instructional improvement. 

3. The faculty member must believe that collecting and 
using student data are feasible within the constraints of the 
situation, both personal and contextual. 

4. The faculty member must believe that the benefits of the 
gathering and use of student data out weigh the amount of effort 
required to follow through with the process (though effort was 
not yet included in this study). 

We further believe that the Factors model will apply across 
contexts because similar constructs have been tested on widely 
different outcomes. Connor and Norman (2005a) supported the 
notion that the many theories of behavior choice have “considerable 
overlap between constructs contained in the main social cognition 
models of health behavior” (pg. 16). We would say that these 
similarities exist not just in models of health behavior, but in many 
areas in which humans make choices. Some even span theories. For 
example, Rogers (2003) diffusion of innovation theory highlighted 
characteristics of an innovation and those who adopt it. Among those 
qualities of the innovation listed are relative advantage, compatibility 
with the adopter’s beliefs, ease of use, and observability of the 
outcome. Bourrie, Cegielski, Jones-Farmer, & Sankar, (2014) mirrored 
this in their similar list of innovation features: I) relative advantage; 2) 
ease to implement; 3) ease of use; and 4) adaptability. As to features 


of the faculty adopters, Bourrie, Jones-Farmer, and Sankar (2016) 
included the familiar qualities of efficacy toward change, support from 
principals (i.e. administrators), benefits from change, attitude toward 
the innovation, along with openness to change, the need for change, 
the appropriateness of the change, awareness of the innovation, 
concern for student outcomes, and motivation. 

The Benefit of a Model of Factors that Affect Faculty 
Use of Student Data. 

We argue that having a conceptual model of factors that influence 
faculty use of student data has theoretical benefits as just discussed. 
But more important, it can highlight areas where those working with 
faculty can design programs that will support positive factors and 
minimize negative ones. For example, if faculty self-efficacy is a key 
factor, then programs should incorporate components that increase 
or support self-efficacy of faculty. One approach is the use of other 
faculty who were successful at data use acting as mentors to show 
doubters what can be done.This value of mentors is exemplified by 
the faculty learning communities approach to change. For another 
factor, ease of use, the importance of making complex student data 
such as learning analytics easy to use and interpret for faculty has 
been discussed by the leading thinkers in the field (Dyckhoff, Lukarov, 
Muslim, Chatte, & Schroeder, 2013; Macfayden & Dawson, 2012; 
Siemens, 2012). Innovations that produce highly effective, yet simple 
implementation of change would be of great value to the faculty 
member who is interested in improving student learning. 

FUTURE RESEARCH 

There will continue to be various versions of the Factors model 
that will arise. Some extensions of the work reported in this paper 
are needed, such as a need to have the study repeated, this time to 
completion, to allow all the variables to be examined over a much 
longer time line. Change does not come easily or quickly. 

It would be helpful for the field to create some widely accepted 
construct definitions in order to develop instruments that can be 
generalized. Measurements not hampered by the disadvantages of 
self-report that are easy to deploy and easy to understand would 
be particularly useful. This is a caution to the learning analytics 
community (Dyckhoff, et al., 2013; Macfayden & Dawson, 2012; 
Siemens, 2012), in which analyses and presentations of data often rely 
on very complex models. 

Finally, faculty themselves should become more familiar with 
educational research. We look to programs like SOTL and the 
support of the Carnegie group to continue to lead the way, as they 
have so effectively up to this point. Faculty are key stakeholders and 
implementers of change in education. Without their support the 
best technology, the best information, the best data, and the best 
innovations will die on the vine. With their support, really innovative 
growth in education is impossible to stop. 
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